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Kenneth B. Hoyt 

Director, Office of Career Education 
United States Office of Education 

Introduction 

Concerns relative to evaluating the effectiveness of career education have 
been prominent since inception of the movement in 1970. Early attempts to 
summarize and synthesize the literature on this subject are illustrated by reports 
prepared by Tuckman and Carducci 1 for the National Institute of Education in 197<t and 

by Development Associates for the Division of Vocational and Technical Education, 

2 
U.S. Office of Education in 1975. Since OE's Office of Career Education was created 

by Congress in I97f, several additional formal attempts to summarize and report 
on studies concerned with evaluation of career education have been published. These 
Including one by Enderleln in 1976, one by Bhaerman'in 1977, and one by Bonnet 5 
In 1978. 



Tuckman, B. W. & 3. A. Carducci Evaluating Career Education! A Review and Model. 
Report No. 36. National Institute of Education (DHEW). Washington, D.C.: 
U.S. Government Printing Office, 197^. 

2 

Development Associates, Inc. Evaluation of Vocational Exemplary Projects! Part 
D, Vocational Education Amendments of 1968 A Prepared for the Division 
of Vocational ancTTechnical Education, U.S7Off ice of Education, March, 1975. 

Tlnderlein, Thomas A Review of Career Education Evaluation Studies. Washington, 
D.C.: U.S. Government Printing Office, 1976. 

Bhaerman, Robert D. Career Education and Basic Academic Achievement! A 
Descriptive Analysis of The Research. Washington, D.C.: U.S. Government 
Printing Off ice, 1977. 

Tionnett, Deborah G. A Synthesis of Results and I Programmatic Recommendations 
Emerging From Career Education Evaluations In 1975-76. Report prepared 
under Grant Number G007604329, Office of Career Education, U.S. Office 
of Education. Washington, D.C.s U.S. Government Printing Office, 1978. 
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In addition, OE's Office of Career Education has had prepared, under contract, 
two monographs - one by Mitchell and the second by Bonnet 7 - designed to provide 
models for use by those involved in evaluating K-12 career education efforts. Further, 
OE's Off ice of Career Education has published two special monographs, one concerned 

with summarizing recommendations of invited experts on evaluation of career educa- 

s 
tion and the second summarizing presentations on evaluation of career education 

made at the 1976 Commissioner's National Conference On Career Education 9 . 

It seems safe to say that never has such a small program, within the United 
States Office of Education, devoted so much attention to evaluating its effective- 
ness on such a sustained basis as has been true for career education. Further, it 
also seems safe to say that, taken as a whole, the available evidence is more positive 
than negative - i.e., at present, it seems safer to say that career education can be eifective 
than to sav it cannot be effective. This fact will be illustrated by reports to be pre- 
sented in this monograph. 

The interest of the Congress in the evaluation of career education continues 
and is easily illustrated through study of P.l. 95-207, CAREER EDUCATION INCEN - 
TIVE ACT passed by Congress on December 13, 1977. Sec. 5(a)(2)(C) of that act 



Mitchell, Anita M. Ways to Evaluate Different Types Of Career Education Activities; 
A Handbook of Evaluation Models. Palo Alto: American Institute for Research, 
T97JT 

Bonnett, Deborah G. Evaluation Design and Reporting jnCareer Education. Wash- 
ington, D.C.: U.S. Government Printing Office, 1978. 

T-loyt.K.B. Perspectives On The Problem Of Evaluation In Career Education. Wash- 
ington, D.C.; U.S. Government Printing Office, 1976. 

TJatta, Lois-ellin et al Career Education; What Proof Do We Have That It Works? 
Washington, D.C.: U.S. Government Printing Office, 1977. 



calls for reserving an amount not to exceed one-half of one percent of funds ap- 
propriated by Congress for purposes of carrying out a national evaluation of the 
effectiveness of career education programs receiving funds under the Act. 
Sec. 7(5) of the Act requires each State Education Agency receiving funds to pro- 
vide proposed criteria to USOE for evaluating the extent to which the State wiJl 
achieve its objectives under the State Plan. Sec. 8(a)(2)(D) allows the State Educa- 
tion Agency to use its State leadership funds under the Act for purposes of conduct- 
Ing statewide needs assessment and evaluation studies. Sec. 8(a){3)(M) allows local 
school districts receiving funds under the Act to use such funds, in part, for conduct- 
ing needs assessments and evaluations. It is clear that, as we move into the decade 
of the 1980s, the Congress desires that the emphasis on evaluation of career educa- 
tion apparent during the decade of the 1970s be continued. 

One of the major problems facing career education at the present time Is that 
most of the literature directly related to evaluation of career education has not 
yet found Its way into referred journals and/or books. Instead, much of it Is in the 
form of project reports submitted to the U. S. Office of Education and documents 
published as monographs by the U.S. Government Printing Office. Such reports 
tend to become part of the "fugitive literature" not readily accessible to those in- 
terested in the problem. Thus, the need for this monograph is seen, in part, as stemming 
from a desire to summarize some of the earlier published documents in one place 
in order to try to avoid the "fugitive literature" problem to the greatest possible 
extent. 

A second need leading to production of this monograph is the need to reflect 
on experiences in evaluating career education during the decade of the 1970s In 
hopes that efforts conducted during the coming decade may profit from such ex- 
periences. Thus, the contents of this monograph are divided into two major sections. 



In the first part, an attempt will be made to provide brief reviews of some of the 
major publications referred to earlier that have been prepared for and/or published 
by the United States Office of Education. The second major part of this monograph 
considers a number of problems and promises facing those who attempt to evaluate 
career education in the decade of the 1980s. 

Summary of Previously Reported Evaluation of the Effectiveness 
Of Career Education Efforts At The K-12 Level 

As of the time this monograph is being written, limited copies of each of the 
documents to be reviewed in this section are still available. For those having access 
to such documents, it will obviously be desirable to study their entire 
contents carefully. Here, only very brief summary statements will be presented 
of what appear to be the critical content of each document along with brief re- 
actions to that content. Taken together, they represent most of the data currently 
available within OE's Office of Career Education pertinent to evaluation of K-12 
career education efforts. 

Purposely omitted from this review are articles published in regularly published 
professional journals. It is assumed that Interested readers can, if they choose, 
locate such articles for themselves. Also omitted from this summary are separate 
reports of doctoral dissertations concerned with the topic of evaluation of career 
education. These, too, are obviously available to interested persons who wish to 
locate them through such standard works as Dissertation Abstracts. 

Bhaerman, Robert D. Career Education and Basic Academic Achievement! A 
Descriptive Analysis of The Research. Washington, 
D.C.! U.S. Government Printing Office, 1977. 

The most comprehensive summary of efforts to evaluate career education 
in terms of its effectiveness In improving basic academic achievement was produced 
by Dr. Robert D. Bhaerman during the 1976-77 period during which Dr. Bhaerman 



worked In OE'sOffice of Career Education. This activity Included an attempt to 
discover and analyze all such attempts that had taken place during the 1970-75 period. 
A total of 38 studies are identified and reviewed in the Bhaerman monograph. Of 
these, two were doctoral dissertations, seven (including one involving only one 
teacher) were from local K-12 school districts, eleven from county K-12 school 
systems, two from Statewide studies, two from the "Partners In Career Education" 
series of studies conducted in Texas, four from OCE funded career education demon- 
stration projects, and the remainder from evaluation studies related to Experienced 
Based Career Education (EBCE) conducted under auspicies of various regional R&D 
laboratories. The search for reports of evaluation studies for inclusion in this mono- 
graph included an ERIC search as well as access to other reports that had been sent 
to OE's Office of Career Education during the 1974-76 period. 

Nature of the i Studies! ; General Summary Statements. The majority of these 
38 studies focused on mathematics (28 studies) and/or reading (23 studies). However, 
data were also reported on language arts (1ft studies), studies skills (ft studies), social 
studies (3) and science (2). Apparently, the term "basic academic achievement" 
has been used to describe each of these subject matter areas. It is obvious that, 
most commonly, it refers to reading and mathematics. Locations in which these 
studies were conducted were found throughout the Nation with the greatest number 
coming from the Northeastern States (17 studies) and the fewest from the far-Western 
States (ft studies). The use of pre-tests and post-tests were seen in 25 of the 38 
studies with the use of control as well as experimental groups found In 13 studies. 
Measurement Instruments were, for the most part, nationally used norm referenced 
tests of basic academic achievement. Sample sizes varied from as few as 5 pupils 
to as many as several hundred. In terms of grade levels included, all grades, K-12 
were found with most of these studies including pupils In Grades 3 - Grade 8. 
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Llmltations of the Studies; Summary Statements. Bhaerman points out the 
most commonly cited limitations given by authors of the reports he reviewed as 
including! (a) problems relating to establishing control and experimental groups; 
(b) problems relating to procedures for implementing the career education program 
(e.g., "students did not receive a concentrated exposure 11 )) (c) problems related to 
testing procedures; and (d) identified problems related to the short time period covered 
by the project. In addition, in a list of "general observations" found in the monograph, 
Bhaerman lists the following which could also be taken to represent some general 
limitations associated with the types of studies he analyzed; (a) there was no apparent 
common operational meaning for the term "career education"; (b) a detailed description 
of the career education "treatment" was often not clearly stated; (c) the scope of 
the studies varied widely; and (d) the quality of the reports also varied. 

The Career Education "Treatment". While some of the studies Bhaerman reviewed 
apparently did not specify the career education "treatment" with any degree of 
clarity or exactness, those that did so led him to make the following generalizations 
regarding the nature of that "treatment": (a) it is a longitudinal effort beginning 
in the early elementary school and continuing throughout the formal education 
system; (b) It emphasizes the process of career development; (c) over half of the 
studies indicated the development of specific curriculum guides, units, learning 
modules or lesson plans for use in "infusing" career education concepts into regular 
classrooms; and (d) staff development for teachers was a common element in many 
of these studies as were use of community resources, field trips, site visits, and 
"hands on" experiences. The majority of these studies covered one academic year in 
length, but the range was large with some projects involving 2-3 years and others 
periods as short as 6-12 weeks. Bhaerman also reported that there seemed to be 
a tendency for those career education "treatments" described most fully as comprehensive 
efforts to be associated with more statistically significant findings whereas those 
studies reporting minimal and/or negligible effects of a career education "treatment" 
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also tended to report the treatment itself as "minimal." 

In studying the detailed tables Bhaerman included to present a descriptive 
analysis of each study, it is most difficult to tell the extent to which the kind of 
career education "treatment" outlined in the OE Monograph A Primer for Career 
Education was followed, but my general impression was that large parts of that 
"treatment" were missing in most of the studies included in this monograph. On 
the other hand, where "treatments " were described in some detail, the general 
impression one receives is that there is a legitimate basis for calling what was done 
by the term "career education." The prime exceptions, of course, are those studies 
concentrating on the Experienced Based Career Education Model, When this classic 
model is used, the proper term is "EBCE" but it is not "career education." 

Results; Of the 38 studies, 19 reported data indicating that reading and/or 
mathematics was Impacted at the .05, .01, or .001 level of significance favoring 
pupils who had been exposed to career education or, in the case of the EBCE studies, 
that the treatment students maintained their growth in the basic academic skills 
areas. Sixteen studies were moderately supportive in that they reported data which 
either indicated reading and/or math grade equivalent score improvement, achieve- 
ment gains for both experimental and control groups, or varied Impact. Three studies 
reported data which indicated either negligible or minimal impact. 

Of the 19 studies Bhaerman labels as "generally supportive," the apparent 
meaning of that term Is that, on at least one of the areas of basic academic achieve- 
ment included, statistically significant pre/post and/or control/experimental gains 
were registered for pupils exposed to the career education "treatment." Note that 
Bhaerman referred to such studies as "generally supportive" in spite of the fact 
that, within some of these studies, some of the comparisons failed to produce a 
positive effect for the career education "treatment." Of the 16 studies Bhaerman 
labels as "moderately supportive," that term appears to indicate that, in general, 



while the career education treatment appeared to show some positive gains, such 
gains were typically not greater than those observed among control group pupils 
- i.e., that career education didn't lower basic academic achievement over what 
would have occured had no such treatment been present. The three studies Bhaerman 
labels as "unsupportive" were ones where no impact was seen. 

Conclusions; In his "conclusions" section of this monograph, Bhaerman makes 
several generalized statements. The following quotations appear to represent the 
conclusions he reached: 

"...the Data Charts disclose that the experimental groups did not con- 
sistently outperform their counterparts. Yet, the overall data is reassuring. 
Looking not at one isolated report, but viewing the total picture, one can 
observe' the situation as quite favorable. There are some who have felt 
that career education would hurt academic achievement. That is certainly 
not an interpretation this analyst would draw from the data." 

"The overall observation I would make is that academic achievement 
generally was either impacted positively or, when it was not, it did 
not interfere with that goal....I believe the general directions are 
positive and the overall tendencies are clear." 

My own conclusions, based on study of the "Data Charts" which appear in the 
Bhaerman monograph, lead me to concur fully with his statement that the evidence 
he summarized certainly makes clear the fact that the kinds of career education 
"treatments" represented in these studies give no indications that career education 
acts to depress academic achievement - i.e., as Bhaerman says, "it doesn't hurt 
academic achievement." However, when 1 look closely at the "Data Charts" for 
those studies Bhaerman labels as "generally supportive," I find several where, in 
spite of the fact the career education "treatment" produced statistically significant 
differences in academic achievement of pupils in some areas, it failed to do so in 
others. There is no way of telling, with exactness, how many times the results were 
statistically significant and how many times they showed no statistically significant differences 
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between pupils who were exposed to the career education "treatment" and ouoils who were 
not. Because of this, we do not know the extent to which those statistically signifi- 
cant differences reported are due to chance and how much they were really due 
to the career education treatment. Thus, these studies, while certainly promising, 
do not appear in any way to be conclusive. 

In short, it seems to me these data tell us with a reasonable degree of certainty 
that a career education treatment has not lowered basic academic achievement , 
but they do not tell us, with an equal degree of assurance, that the career education 
treatment will improve basic academic achievement. The best they appear to be 
able to tell us is that the career education treatment can - and sometimes does - 
serve as a vehicle for improving basic academic achievement. 

If the data found in the Bhaerman monograph were to be interpreted in the 
most negative way, one could say that they leave open the question of the probable 
impact of a career education "treatment" on academic achievement. That is, findings 
showing no statistically significant differences between experimental and control 
pupils obviously may result from any one of three basic conditions including: (a) there 
really are no differences; (b) the time period and/or experimental conditions may 
have been inadequate to demonstrate that the experimental pupils actually do out 
perform the control pupils; or (c) the time period and/or experimental conditions 
may have been inadequate to demonstrate that the control pupils actually out perform 
the experimental pupils. It is, of course, the third possibility that must concern us. 
It is very difficult to demonstrate, in a clear and convincing fashion, that any kind 
of '.'new" approach to the teaching/learning process actually improves academic achieve- 
ment. The typical finding is one of "no statistically significant differences." Typically, 
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a time period extending over several years is required in order to demonstrate clear 
and conclusive results. These studies, by and large, did not cover a time period 
of over one year. Thus, we leave those who are "disbelievers 11 with as much basis 
for claiming that, in the long run, career education will be shown to have a negative 
effect on academic achievement as do the "believers" for claiming that, in the long 
run, it will have a positive effect. 

More than anything else, these data demonstrate to me the absolute necessity 
for finding and/or conducting a series of solid evaluation studies using randomly 
assigned experimental and control pupils under conditions of pre/post testing where 
assurances can be given that the kind of "career education treatment" recommended 
in A Primej: For Career Education is actually applied for a period of years with annual 
assessments called for. To date, no such comprehensive studies have yet been conducted 
to the best of my knowledge. They are certainly needed. 

Enderlein, Thomas A Review of Career Education Evaluation Studies. Washington, 
D.C.: U. S. Government Printing Office, 1976. 

While serving as an intern in OE's Office of Career Education during 1976, Dr. 
Thomas Enderlein was asked to compile a summary of available literature related to 
evaluation of K-12 career education efforts that had taken place prior to initiation of 
demonstration projects funded under provisions of P.L. 93-380, Section 406 (which 
began during the year Dr. Enderlein made his search). His review is divided into three 
major sections; (a) career awareness; (b) career decision making; and (c) academic 
achievement. Since the studies reviewed by Enderlein in the domain of the effect of 
career education on increasing academic achievement are included in the set of studies 
found in the Bhaerman monograph summarized earlier, they are eliminated from this 
summary. 
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se impressive summaries of data found in the Enderlein monograph illustrate 
3m of "fugitive literature" facing those who attempt to provide a compre- 
jmmary of the literature related to evaluation of career education. These 
orts are referenced in the Enderlein monograph as follows: 

kman, B.W. & 3.A. Carducci Evaluating Career Education! A Review 
And A Model. Report No. 36. National Institute of Education (DREW). 
Washington, D.C.: U.S. Government Printing Office, 197t. 

'elopment Associates, Inc. Evaluation of Vocational Exemplary Projects; 
Part P. Vocational Education Amendments of 1968. Prepared for the 
Division of Vocational and Technical Education, U.S. Office of Educa- 
tion, U.S. Office of Education, March, 1975. 

lavioral Research Associates Cochise County Career Education Project, 
Evaluation Report, 197t-75. Tucson, Arizona, 1975. 

:hese three reports is readily available today to those seeking original data. 
, the Enderlein monograph itself may well also become largely unavailable 
few years. For these reasons, it seems appropriate here to reproduce the 
les found in the Enderlein monograph from each of these three reports. 
ear here exactly as in the Enderlein monograph. 
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TABLE 3 



COMPARISONS OF STUDENTS WITH HIGH AND LOW EXPOSURE TO CAREER 
EDUCATION ON SEVERAL DIMENSIONS RELATED TO THE ARIZONA CAREER 
EDUCATION MATRIX 



High Exposure Low Exposure 

Sample Sample 



t-test Significance 



Secondary Students 


Educational Awareness 


46.8 


44.3 


3.01 


0.001 


Knowledge of Skill Req. 


54.5 


49.6 


2.40 


0.01 


Knowledge of Factors Con- 










tributing to Job Satisfac- 










tion 


63.8 


55.7 


3.40 


0.001 


Common Threads in Jobs 


68.2 


57.0 


4.64 


0.001 


Economic Awareness 


53.1 


48.5 


1.83 


0.04 


Awareness of Career Mobility 


40.9 


34.0 


2.78 


0.003 


Awareness of Factors Influ- 










encing Occup. Structure 


53.2 


47.7 


2.27 


0.002 


Decision Making 


35.3 


30.2 


2.94 


0.001 


Employability Skills 


26.6 


24.4 


3.04 


0.001 


Appreciations and Attitudes 


45.3 


37,0 


3.90 


0.001 


Knowledge of Career Clusters 










Score 


34.2 


30.8 


1.85 


0.04 


Interest in Career Clusters 










Score 


25.9 


24.7 


1.97 


0.025 


Elementary/Intermediate Students 


Educational Awareness 


50.1 


48.5 


1.86 


0.05 


Knowledge of Skill Req. 


59.4 


50.5 


3.45 


0.001 


Knowledge of Factors Con- 










tributing to Job Satisfac- 










tion 


54.6 


48.5 


2.57 


0.005 


Common Threads in Jobs 


65.7 


59.8 


2.12 


0.01 


Economic Awareness 


58.2 


52.6 


2.24 


0.01 


Decision Making 


61,5 


54.7 


2.67 


0.005 


Appreciation and Attitudes 


60.2 


54.9 


2.01 


0.020 


Knowledge of Career Clusters 










Score 


49.9 


44.8 


2.28 


0.01 
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In addition to the data reproduced above, the Enderlein monograph provided 
brief summaries of findings - almost all of which were highly positive - from several 
other studies evaluating the effectiveness of career education. However, Enderlein's 
summary of findings failed to provide any detailed information regarding the specific 
career education "treatment" that was involved, the nature of the experimental de- 
sign, or specific findings and/or conclusions. For these reasons, no attempt Is made 
here to include yet another table summarizing such studies. The following quotes 
from the "Summary" found in the Enderletn monograph provides a general 
flavor with respect to the studies included In his revlewi 



",,..In general, students who participated in career education projects achieved 
gains in occupational information, attitudes toward work, career maturity, 
knowledge of the world of work, and reality about their career planning. 
Findings relating to the effects of career education upon career decision- 
making were similarly positive: students who had participated In career 
education projects at both the elementary and secondary levels were better 
able to make decisions which required analyses of their abilities, needs and 
interestsj of occupational rolesj and the relationship of self to career plans.... 
Overall, the evaluation efforts to date display positive results In the many and 
varied aspects of career education." 

Certainly, the brief descriptions found in the Enderieln monograph appear to be sup- 
portive of the general conclusions quoted above. However, because the actual studies 
themselves were unavailable for review at the time this monograph was written, 
it seems inappropriate to comment further on the contents of the Enderlein 
monograph, 
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Datta, Lols-ellin et al Career Education; [What Proof Do We Have That It Works? 
Washington, D.C.t U.S. Government Printing Office, 1977. 

At the Commissioner's National Conference on Career Education held in Houston, 
Texas in November, 1976, a major theme session was conducted using the title of 
this monograph. The monograph itself contains the formal statements made by presenters 
at this theme session. 

The monograph begins with an introductory paper written by Dr. Lois-ellin 
Datta of the National Institute of Education. This paper is devoted primarily to a 
series of short critiques of the remaining papers presented, The concluding portions 
of the paper concentrate attention on some of the major philosophical/policy impli- 
cations associated with efforts to evaluate career education. It is a very thoughtful 
paper and one that deserves much wider attention than it has apparently received 
to date. 

The second paper, written by Dr. Elvis Arterbury, reported on two studies of 
the effectiveness of career education at the fifth grade level - one concentrating 
on the effectiveness of career education in Increasing achievement in Language Arts 
and Social Studies and the second asking this same basic question with respect to 
academic achievement In mathematics. The Arterbury paper produced generally 
favorable evidence for both of these studies indicating that a career education 
"treatment" can produce significant gains in basic academic achievement among 
elementary school pupils. The Datta critique of this paper noted that these studies 
failed to distinguish that part of the "treatment" that associated itself with infusing 
career education concepts from that part associated with the instructional content 
itself. This is a major problem associated with a career education "treatment" oriented 
primarily around an Infusion emphasis. That problem will be examined in some detail 
later in this monograph. 
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The paper written by Dr. Frank Rapley for this monograph described his attemp 
to evaluate a K-12 career education effort In the Jefferson County, Kentucky School 
System. While, due to lack of complete data due to problems associated with imple- 
menting a desegregation order, he was unable to perform all the statistical analyses 
he had hpped for, the data presented in this paper show gains in both reading and 
mathematics achievement for pupils exposed to the career education treatment. 
The most fascinating portions of this report are seen by studying the approaches 
being tried to evaluate a total K-12 career education effort rather than simply portio 
of that effort. The Datta critique of this paper warned readers not to make strong 
inferences concerning the worth of career education due to the weaknesses In the 
experimental design made necessary by the school reorganization problem. It also 
commended this paper to those interested in a total K-12 approach to evaluating 
career education. 

One of the clearest - and best - examples of a report evaluating career educatU 
is seen in the paper prepared by Dr. Phil Spleth of the Dade County, Florida Public 
Schools. Dr. Spieth's description of the career education "treatment" is both clear 
and complete. While the particular 'treatment" being described will not be one that 
all school systems will want to endorse, at least the description of what was done 
in the name of "career education" is made clear. While differences between experl - 
mental and control pupils reported in this study were not large, they were uniformly 
in directions favoring those pupils who had been exposed to career education. The 
Datta critique of this paper applauded the use of trend data over a period of years 
which Dr. Spieth had used. The clearest findings in this study were those indicating 
that the most favorable results are associated with those career education efforts 
that have been in operation for the longest period of time - thus emphasizing, once 
again, the importance of evaluating career education in a longitudinal design. 
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Dr. Richard Ruff, Arizona State Department of Education, presented a paper 
for this series summarizing major Statewide findings from Arizona K-J2 school 
systems relative to evaluation of career education efforts during the 1971-75 
period in the State of Arizona. This paper indicated that students with "high ex- 
posure," as opposed to "low exposure," to career education: (1) have a greater 
knowledge of the wide range of occupations available to them; (2) have an increased 
awareness and understanding of educational requirements for various careers; 
(3) have a better understanding of skills and abilities required for various careers) 
and (it) have an increased understanding of economic rewards associated with 
various careers. In critiquing this paper, Datta pointed to the need to assess in- 
dependently the intensity and the quality of career education efforts - i.e., to 
separate out evaluations of "how much" from evaluations of "how well." 

Finally, Dr. Sidney High presented a paper summarizing results from a small 
number of evaluation studies, each of which produced positive results indicating 
the worth of career education. The prime "message" found in Dr. High's paper 
is that career education CAN work - in spite of the fact that many things currently 
being done under the name of "career education" are so weak as to almost surely 
fail to work. Of these studies (1) three studies show evidence of improved academic 
skills; (2) five of improved work values; (3) four of improved career decisionmaking 
skills; (it) two of improved occupational and interpersonal skills; (5) fourteen of 
greater knowledge of educational/vocational opportunities; and (6) one indicating 
that career education provided students with an increase in the sense of control 
over their lives. Datta's critique of this paper reminded us that, positive as these 
data are, they will certainly not suffice to convince the true skeptic of the worth 
of career education - due, in part, to the fact that only short-term outcomes of 
career education were measured. 
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Bonnet, Deborah G. A Synthesis of Results and Programmatic Recommendations 
Emerging From _ Career_Educatlgn Evaluations in 1975-76. 
Washington, D.C.: U.S. Government Printing Office, 1978. 

During 1976, OE's Offfice of Career Education awarded a grant (G007604329) 
to New Educational Directions, Inc. whose purpose was to synthesize and summarize 
evaluation results available from the 108 K-12 career education demonstration 
projects funded by the U.S. Office of Education during the 1975-76 academic 
year. A total of 47 of these programs were in their third year of funding under 
provisions of Sec. 142{c), Part D of P.L. 90-576 while 61 were operating as one 
year projects funded under provisions of Sec. 406, Title IV, P.L. 93-380. Of these 
108 projects, a total of 45 - including 26 Part D and 19 OCE (sec. 406) - contained 
evaluation data judged suitable for use in this project. These 45 studies were 
selected using the following critiera: (a) the report provided sufficient information 
about the measurement instrument for determining which outcome the study 
addressed; (b) the report indicated which of four grade-level groupings the students 
represented; (c) the outcomes of the career education group were compared to 
some reference group; and (d) the report indicated the direction of differences 
between means. There was no requirement that results be analyzed Inferentially 
nor that the number of students, the sampling plan, or the specific student career 
education activities be identified. 

The unit of analysis used in reporting results was called the "outcome study." 
An "outcome study" was arbitrarily defined as the assessment of one learner out- 
come objective at one grade level. For example, an evaluation investigating a 
K-12 program's impact on reading achievement, work attitudes, and career knowledge 
at each of Grades 4, 8, and 12 would, using this definition, consist of a total of 
9 "outcome studies." Using this definition, a total of 500+ "outcome studies" were 
classified under one of the 10 "learner outcomes" found in the OCE monograph 
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entitled Perspective On The Problem of Evaluation In Career Education. Thus, 
a separate section of this report appears for each of these 10 "learner outcomes." 

Treatment; Bonnet was unable, with the data available to her, to specify with 
any kind of exactness the particular kind(s) of "career education treatment" that produced 
the results reported in this monograph. As common traits, she listed the facts 
that all programs: (a) involved the development of materials, training of school 
staff, and involvement of the community) (b) shared the goal of preparing individuals 
for participation in the world of workj and (c) appeared to have embraced the 
basic definition, philosophy, and concepts of career education expressed in the 
1975 USOE policy paper, An Introduction to Career Education. Within these broad 
common elements, she discovered very wide variation to exist in such important 
treatment elements as: (the proportion of students who had received whatever 
career education "treatment 11 was being offered; (b) the length of time the "treat- 
ment" had been applied: (c) the definition of "Infusion" used in specifying the "treat- 
ment"} and (d) the extent' to which elements in addition to "Infusion" within regular 
classrooms were included in the definition of the "career education treatment." 

Thus, while the results reported in this monograph can be said to reflect a 
career education "treatment" with some common phjlosophical meaning, they 
cannot be said to reflect a career education "treatment" with some common 
operational activities meaning. Apparently, the best that can be said is that some 
bonofide career education Implementation strategies were used, but it is not 
known which strategies produced any given reported result at any specified level 
or length of application. The results reported in this monograph must be viewed 
given this severe restriction of meaning with respect to the exact "career 
tion treatment" that was used. 

Design; The intent of this project was to synthesize evaluation results related 
to each of the 10 OCE "learner outcomes" as found In the 500+ "studies" identified 
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by the author. A necessary first step was to make subjective judgments regarding 
the extent to which the various measurement instruments used related to one 
of these 10 "learner outcomes." That is, no attempt was made to infer that tte 
measurement instruments were designed to be direct measures of each "learner 
outcome." For some, the relationship is quite obvious while, for others, it is not. 
For example, there were fairly direct measures used to assess "knowledge of oc- 
cupations" but, in lieu of direct measures of "self-understanding," measures were 
used intended to measure "self esteem." 

For each of the 10 "learner outcomes," the following data were gathered and 
reported: (a) number of projects assessing that outcome ; (b) the approximate 
numbers of students involved in assessments; (c) the number of "studies;" (d) the 
grade level categories (K-3, 4-6, 7-9, 10-12) used; (d) the type of research design 
used (e.g., "pre-post without comparison," "post with comparison"),- (e) the number 
of "studies" with positive results significant at the .05 levelj (f) the number of 
studies with positive differences between means; and (g) the probability that positive 
and negative differences are equally likely. 

Further, the design called for a table specifying each of the measuring instru- 
ments used, the number of times the instrument was used at each of the grade 
levels, the number of times statistically significant results were obtained, and 
the number of times it produced positive differences between names. Those data 
are reported in an Appendix of this monograph. 

The 10 OE "learner outcomes" were, in some instances, sub-divided into various 
"sub-outcomes" where, in the opinion of the author, this was justified. The following 
represent the "learner outcomes" used as major categories for summarizing results 
of the 500 + "studies;" 



Learner Outcome //1-a; Competent in the basic academic skills required for 
adaptabiiity in our rapidly changing society (READING ACHIEVEMENT) 

Learner Outcome ffl-b; Competent in the basic academic skills required for 

adaptability in our rapidly changing society (MATHEMATICAL ACHIEVEMENT) 

Learner Outcome tf 2; Equipped with good work habits (ATTITUDES TOWARD 
SCHOOL) 

Learner Outcome //3: Equipped with a personally meaningful set of work values 
that foster in them a desire to work. 

Learner Outcome #-ai Equipped with CAREER DECISION-MAKING SKILLS 

Learner Outcome ffH-bt Equipped with JOB-HUNTING and JOB GETTING 
SKILLS 

Learner Outcome ff5-a; Equipped with JOB SPECIFIC OCCUPATIONAL SKILLS 
and interpersonal skills at a level that will allow them to gain entry into 
and obtain a degree of success in the occupational society 

Learner Outcome //5-bj Equipped with job specific occupational skills and 
INTERPERSONAL SKILLS at a level that will allow them to gain entry 
into and attain a degree of success in the occupational society 

Learner Outcome //6-2; Equipped with a degree of SELF-UNDERSTANDING 
and understanding of educational-vocational opportunities sufficient for 
making sound career decisions 

Learner Outcome //6-b; Equipped with a degree of self-understanding and 

UNDERSTANDING OF EDUCATIONAL-VOCATIONAL OPPORTUNITIES 

sufficient for making sound career decisions 

Learner Outcome_#7i Aware of means available to them for continuing and 
recurrent education 

Learner Outcome //Si Either placed or actively seeking placement in a paid 
occupation, in further education, or in a vocation consistent with their 
current career decisions 

Learner Outcome //9; Actively seeking to find meaning and meaningfulness 
through work in productive use of leisure time. 

Learner Outcome fflOi Aware of means available to themselves for changing 
career options - of societal and personal constraints impinging on career 
alternatives 

Results; A "bird's eye view" of the extensive results found in this monograph 
are summarized, with respect to each of the "Learner outcomes," in the table 
which appears below. 



~ ~ Table i Summary of Results Obtained from 500+ "Studies" 

Reported in the Bonnet Monograph 

% "studies" with % "studies" with Probability that positive 

rSr 1 " K^TeST- aimar 1 " 

At .UJ IGVcJl __ ~~ - * ~ ~~~~ 



io(v (fl% NS 

l~ a l ;-!. NS 

1 u 39% 5* 

% <f(f% NS 

2 ||J 6 g % Less than .02 (+) 

3 In % 70% Less than .001 (+) 

l~h Only 6 studies reported - one with statistically significant findings _ 

5lg Not measured by career education. Perceived as a vocational education 

?L blerr " 61% Less than ,05( + ) 

tm 9% 57% NS 

6-a (I) ' ,..,, MC 

6 b (2l 15% 58 

," nniv I study reported - it was positive 

I TOO EARLY TO EXPECT CONCLUSIVE DATA REGARDING THIS OUTCOME 

q Only 1 study reported - it was positive 

10 Only ' studies reported - 3 of which were positive 

In addition to the ]<inds of summary statistics illustrated in the table above, 
results are also summarized, in more detailed form, for evaluation studies associated 
with each learner outcome goal that produced positive results. An example of 
this kind of reporting can be seen in the following quotation referring to one of 
the studies evaluating the effectiveness of a career education infusion approach 
in the classroom on Improvement of basic academic skills: 

"Perhaps the most encouraging results totte^^faKfaVfe^Jj^ 

f\ Pa i t to Efficulum e involved^ ftQf teachers and affected ALL students 
n the school system. In the spring of 1973 and again In the spring of 1976, 
aH of the app oxiniately 350 students in each of grades 3, 6, 8, and 9 were 
admin Isterac I the reading and mathematics subtests of the appropriate levels 
ofThe CaJiforn a Achievement Test. In reading, system-wide average scores 
d rowed a Si but significant amount at grades six and nine but improved 
aSiiffcant H grades three and eight. The improvement at third grade 

S IS*- , r-,Fir with the 1976 third-graders' mean raw score (60.3) thirteen 
aerc'en hlg fhan tTa of 1973 third-graders (53.3). Mean scores in math 
Coved at a 1 our grade levels. Third, sixth, and eighth grade improvements 
w2?e atatls Ically significant and Impressively iarge-thlrteen percent at third 
grade twelve p ^ cent at sixth grade, and twenty-two percent at eight grade." 
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A total of R career education demonstration projects are referenced specifically 
one or more times with descriptions of their results using a format similar to 
that given in the example above. 

Conclusions! Bonnet includes a 1-2 page discussion/conclusions section 
for each of the 10 learner outcomes. Near the end of the monograph, she also 
provides a 1-2 sentence conclusion statement with respect to each learner outcome. 
These 1-2 sentence "conclusions," as given by Bonnet, are reproduced below: 

Career education seeks to produce lndiyidualsjwhp f when they LEAVE school 
(ajt any age or at any jevel) are; 

1. Competent in the basic academic skills required for adaptability In our 
rapidly changing society 

CONCLUSION: Overall, results are neither positive nor negative but in 
selected instances improvements were both statistically 
and educationally significant 

2. Equipped with good work habits 

CONCLUSION: Measured attitudes toward school were affected only in 
Isolated instances. Data concerning work habits are 
largely lacking 

3. Equipped with a personally meaningful set of work values that foster in 
them a desire to work 

CONCLUSION: Good evidence of success has been demonstrated in 
strengthening youngsters' desire to work. 

<f. Equipped with career decision-making skills, job-hunting skills, and job- 
getting skills 
CONCLUSION: Career decision-making skills were strengthened with 

great consistency, Data concerning job-seking skills are 

sparse but somewhat promising 

5, Equipped with job specific occupational skills and interpersonal skills at 

a level that will allow them to gain entry into and attain a degree of suc- 
cess in the occupational society 

CONCLUSION: For several reasons career education evaluations rarely 
evaluate job specific occupational skills. Results 
of interpersonal skills evaluation are encouraging but 
are not conclusive 

6. Equipped with a degree of self-understanding and understanding of educational- 
vocational opportunities sufficient for making sound career decisions 
CONCLUSION: Self-esteem is rarely affected as It is currently measured. 

and no objective data of self-understanding were reported. 
Career awareness was achieved in the overwhelming majority 
of cases and there is reason to believe that the intent 
of the goal Is being met on a broad scale 
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7. Aware of means available to them for continuing and recurrent education 
CONCLUSION: Evaluation results are Jacking 

i 

8. Either placed or actively seeking placement in a paid occupation, in further 
education, or in a vocation consistent with their current career decisions 
CONCLUSION: It is too early to measure career education's influence on 

this outcome on a wide-scale basis, 

9. Actively seeking to find meaning and meaningfulness through work in productive 
use of leisure time 

CONCLUSION: Only one study was reported; its results were promising 

10. Aware of means available to themselves for changing career options - of societal 
and personal constraints impinging on career alternatives 

CONCLUSION: Evaluation results are sparse but very encouraging 

Discussion; The Bonnet monograph represents a valuable addition to the literature 
concerned with evaluation of career education. It appears that she did a sound 
and credible job with the data available to her. Results in this monograph are 
especially pertinent to those making funding decisions with reference to career 
education at the federal level because each project is one funded with federal 
career education demonstration funds during the 1975-76 academic year. It is 
partly for this reason that a thoughtful discussion of the contents of this monograph 
are in order. 

First, it is important to note that the OE "learner outcomes" for career education 
were designed in order that they might be applied to individuals at the point when 
such individuals leave the formal education system after having been exposed 
to a comprehensive K-12 career education effort. Each is obviously not intended 
to be applicable to students at any particular grade level within the K-12 education 
system. Thus, it is not surprising to find that, for the many "studies" Bonnet reported 
in which the participants were elementary school students, no clear results were 
in evidence. 

Second, it must be noted that Bonnet emphasized the measuring instruments 
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avallable to her for use were, by and large, not constructed so as to be direct measures 
of these 10 "learner outcomes." Instead of having such measures available, she was 
forced to use measures which, in her opinion, had some indirect relationship to a 
given "learner outcome." 

Third, it is essential to point out that, in actuality, Bonnet was unable, because 
of the nature of the data made available to her, to specify, with any degree of exact- 
ness, the nature of the career education "treatment" being evaluated. Neither was 
she able to determine how long whatever "treatment" existed had been applied nor, 
In many Instances, whether or not it had been fully applied to ALL students included 
in the studies reported, It must be remembered that, during the 1975-76 academic 
year when these data were being collected, there had yet been no agreement with 
respect to what the "career education treatment" actually includes. There are avail- 
able in her data assurances that some kind of career education "treatment" took 
place - thus justifying the analysis which she did, However, It would be extremely 
dangerous to take Bonnet's findings and conclusions as representing solid generali- 
zations with respect to the worth of these K-12 career education efforts. It was 
simply the best she could do with the data made available to her. Bonnet acknow- 
ledges this in the body of the monograph. 

Fourth, Bonnet's analysis illustrates very well (a) the need for measuring In- 
struments more appropriate for use in evaluating career education's effectiveness, 
(b) the need to insist on sound evaluations for all Federally funded career education 
demonstration projects (note that only 15 of the 108 projects Involved met the minimum 
criteria Bonnet established for Inclusion In her analysis), (c) the need for evaluation 
systems for use in career education that do not depend on criteria related to outcomes 
expected only after a complete K-12 effort, and (d) the need for evaluations to be 
based on the presence of a well defined career education "treatment" that extends 
beyond one year in length. Each of these problems represents a serious current 
challenge for career education. 
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In spite of all the limitations obviously associated with both the data and 
the basic methodology Bonnet employed, her findings, overal], paint a generally 
positive picture with respect to the effectiveness of career education. Certainly, 
the evidence concerning career education's ability to positively affect (a) increases 
in students' desire to work, (b) increases in students' acquisition of career decision- 
making skills, and (c) students' awareness of the nature of occupations and the 
occupational society appear, even under these severe limitations imposed by the 
data available to Bonnet, to have been adequately demonstrated. Further, other 
than in the case of the one learner outcome judged inappropriate for use in evaluat- 
ing career education and the one outcome regarded as inappropriate for use at 
this stage in the history of career education, all other "iearner outcomes" produced 
some statistically significant positive results. In short, while these data do not, 
In any instance, support a claim that career education will produce such positive 
effects, they do support a claim that career education can produce positive effects 
given the right circumstances. The trends are, by and large, positive. At this 
early stage in the career education movement, this is most encouraging indeed. 

The one "learner outcome" for which, to me, the data were most discouraging was 
the one related to career education's effectiveness in increasing students' achievement 
in the basic academic skills. Even here, the data support a contention that a career 
education "treatment" can produce positive results. The problem, I suspect, is two-fold 
here. In part, I strongly suspect that, in many instances, the full career education 
"treatment" (as outlined in the OCE monograph entitled A Primer For Career FH.^ti^ 
was probably not applied. In part, I suspect it is a case of not applying that treatment 
to all pupils for a long enough period of time. Thus, while I am disappointed that these 
data did not appear more positive, I am not discouraged. I continue to remain firmly 
convinced that we will, given the proper application of the career education "treatment- 
be able to demonstrate career education's effectiveness in improving basic academic 
skills - especially where elementary/middle school pupils are involved. 
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Hamllton, Jack A. and Anita M. Mitchell Final Technical Report;. Identification of 

Evaluated Exemplary LActiyUies^ln Career 
Education (K-12). Palo Alto. Californlai 
Am er lean Institutes For Research, 1978. 

This document summarizes activities performed under a contract given the 
American Institutes For Research (AIR) bv OE's Office of Planning, Budgeting, and 
Evaluation. It represents the most comprehensive effort yet undertaken to Identify 
and analyze current attempts to evaluate career education efforts by local K-12 school 
systems. The project called for AIR to identify those career education efforts that 
had been evaluated in a sound fashion, to identify the best of these, and to submit these 
to the Joint Dissemination and Review Panel (JDRP) for review and approval, The 
numbers involved in this AIR effort provide a good basis for judging the current 
"state of the art" with reference to the extent to which sound evaluations exist 
for existing K-12 career education efforts. 

The project began with an attempt to identify existing K-12 career education 
efforts with sound evaluation data. A nationwide search turned up nominations 
of 394 existing K-12 career education sites. Of these, 257 submitted data to AIR 
for review. (It Is, of course, not known whether the remainder did not have what 
they considered to be adequate data or whether they Ignored the AIR request 
for materials). 

Three levels of review were used for judging materials submitted. At Level 
I, materials were evaluated on 7 preliminary criteria including, for example, "the 
evaluation design Includes a comparison group/standard of some kind," and "the 
evaluation report concludes that there is evidence of effectiveness." When submitted 
to this level of review, 64 of the 257 passed. Level II review was much more stringent 
especially in demanding that tests of statistical significance had been conducted 
for student outcome data. Twenty of the 64 sites passed Level II review. In Level 
III, a very stringent and comprehensive set of criteria were applied related to evidence 



-30- 

of effectiveness. Only 1 1 of the 20 sites who had passed Level II review also passed 
Level III. Of those who passed Level III review, 10 were submitted to JDRP and 
7 of those 10 passed, 

A brief capsule summary of each of these 7 projects is presented below in outline 
form: 

1. Akron Career Development Program - Akron Public Schools - Akron, Ohio 

(a) Population! Grades K-10 (60% minority in urban school district) 

(b) Treatment: Infusion In classrooms with strong community involvement 
5<j{ Design: 40 experimental and 40 control classes - end of year testing 
w Resultst Experimental (career education) students outscored control 

students in 35 of 36 comparisons of subgroups at the 3rd, 6th, 8th, 
and 10th grade level in the following cognitive areas: 

(1) knowledge of decision making 

(2) knowledge of "Individual and Environment" 

(3) Knowledge of self 

(4) Knowledge of employablllty and work adjustment skills 

(5) Knowledge of the world of work 

(6) Knowledge of economic understanding 

(7) Knowledge of education and training opportunities 

2. Developmental Career Guidance Project - Pima County, Arizona 

(a) Population: Grades 4-12 pupils In mixed rural/urban setting) 20% minority 

(b) Treatment! Infusion of efforts Into regular classrooms through leadership 
of counselors trained to be career development specialists 

(c) Design: End of year cognitive measures to students in 26 schools with 

high exposure" to career education and 25 schools with "low exposure" 
W Results: Statistically significant differences favoring experimental 
(high career education exposure) over control at each grade level In: 

(1) Career awareness 

(2) Economic awareness 

(3) Career decision making skills 

(4) Employability skills 

(5) Appreciations and attitudes toward work 

(6) Knowledge of career clusters for broad groupings of occupations 

3. Project CAP - Boston Mountains Educational Cooperative - Greenland, Arkansas 

| a j Population! Grades K-8 students In poor rural areas 

W Treatment: Infusion of career education concepts Into regular subject 

matter by regular elementary school teachers 
w Design! Experimental/control classes of students, K-S, with one school 

designated as "experimental" and the other as "control" 
w Resultst Experimental (career education) pupils showed statistically 

significant gains over control students in: 
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(1) Scores on a Career Awareness Test 

(2) Scores on a Career Knowledge Test 

(3) Scores on an Orientation to Career Concepts/Worker 

Activities Scale 

4. Project CDCC - Coloma Community School District - Coloma, 

Michigan 

k) Populations K-6 pupils in both rural and urban areas 

(b) Treatment! 27 career education curriculum units to be 

infused into regular subjects 

(c) Design; Pre/post measures onl 38 career education performance 

objectives 

W Results; Where sustained exposure occured, 95 of 138 performance 
Objectives were reached. Academic achievement in 
basic skills was not reduced and, in some cases, increased 

5. Project CERES - Ceres Unified School District - Ceres, California 

(a) Population! K-6 pupils in rural community 

(b) Treatment! Infusion into regular classrooms; strong community 

involvement 

(c) Design; Experimental/control classes with pre/post tests 

of career awareness and knowledge of careers 

(d) Results; Statistically significant differences favoring experi- 

mental (career education) pupils on cognitive tests 
of "career awareness" and "knowledge of careers." 
Gains in academic achievement for low income 
pupils, but not for middle Income pupils in experimental, 
as opposed to control, groups. 

6. Project Equality - Highline Public Schools - Seattle, Washington 

(a) Population! Grades K-6 pupils - predominantly white middle- 
class 

ft>) Treatment! Development and infusion of 3 sets of materials 
into classrooms designed to reduce effects 
of occupational sex stereotyping in both girls 
and boys 

(c) Design! Experimental/control classes with pre/post testing 

(d) Results! Experimental (career education) pupils showed 

statistically significant gains over controls in 15 
of 17 comparisons. 

(e) Conclusion! Given this treatment, pupils will expand their 

perceptions of job options open to females 
and to males 

7. Project MATCH - Ontario- Montclair School District - Ontario, 
California 

(a) Population! K-8 pupils in urban settings 

( b ) Treatment! Infusion of 110 career education objectives 

into regular subjects 
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(c) Design; Pre/post measures with experimental and controi 

classes randomly assigned at each grade ievel 

(d) Results; Statistically significant differences in cognitive 

items of career development at each grade level 
favoring experimental (career education) pupils 
over control pupils. 

These seven "super" evaluation studies joined, in late 1978, with several 
earlier studies that had previously been approved by the JDRP and submitted 
for inclusion in the National Diffusion Network (NDN) publication of EDUCATIONAL 
PRACTICES THAT WORK. They make a most significant set of additions to the 
literature related to evaluation of career education. 

One of the most valuable portions of this document consists of the list of 
all projects nominated to AIR for consideration in this project. That list provides 
a clear indication that evaluation of career education is a matter of high concern 
on the part of literally hundreds of K-12 school systems involved in implementing 
career education efforts. The fact that most of these have not yet developed 
the kinds of sophisticated evaluation designs that enable them to pass the rigid 
requirements of the JDRP appears to be more a function of the early stages of 
evolution In career education than it does a lack of interest in evaluation. 

In looking at these seven "super" evaluation efforts, it seems worth noting 
that: (a) a majority of them concentrated at the elementary school, rather than 
the secondary school level) (b) most of these positive results were obtained using 
an "infusion" rather than a "special course" approach to implementation of career 
education; and (c) most of these studies used cognitive measures reflecting short- 
term, intermediate goals of career education. These observations hold important 
implications to be discussed in a different section of this monograph. 
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Sumrnarv of the Summaries 

Of the 38 studies reviewed in the Bhaerman monograph, 19 reported statistically 
significant results favoring career education while an additional 16 indicated some 
support for contending that career education is effective. When the three tables repro- 
duced from the Enderlein monograph are examined, one can see positive results reported 
for all 16 studies included in the Tuckman and Carducci report, for 6 of the 13 analyses 
made for 6th and 9th grade pupils in the Development Associates report, and for all 
20 comparisons shown in the Cochise County Arizona career education project. Other 
studies reviewed in the Enderlein monograph but not included here were also generally 
positive, 

If we eliminate from the "Career Education: What Proof Do We Have That It 
Works?" monograph those studies included in other documents summarized in this mono- 
graph, I count a total of 15 additional studies, each of which reported some favorable 
conclusions regarding the effectiveness of career education. 

The Bonnet monograph, because of the unique way in which Bonnet defined 
an "outcome study," cannot be directly compared with the other monographs summarized 
here. The total of 4.5 evaluation studies included in the Bonnet analysis produced a 
grand total of 552 "outcomes," of which 99 were statistically significant at the .05 
ievel of confidence and 352 showed positive differences between means favoring those 
pupils who had been exposed to career education. Of the U studies specifically referenced 
in the Bonnet monograph, all showed positive results demonstrating the worth of career 
education. 

Finally, the Hamilton and Mitchell document shows clear and conclusive evidence 
concerning the worth of career education for the seven enaluation studies that success- 
fully passed JDRP review. That same document clearly illustrates the need for better 
evaluation data In many other projects. 

Taken as a whole, it certainly seems safe to claim that the evidence summarized: 
in this set of monographs points to a conclusion that K-12 career education efforts 
can - and often do - produce favorable results. The overall picture is much more positive 



than it is negative. There are some reasonably good answers that can be given to those 
who inquire with respect to whether or not career education is effective. This body 
of evidence should be sufficient to answer initial questions raised by the true skeptic. 
It will not, however, adequately answer questions raised by the true "disbelievers." 
For such persons, much more remains to be done. 

The persistent difficulties one observes in studying the documents reviewed here 
can be summarized easily. First, there are serious questions, for many of the specific 
studies, regarding exactly what has been tested in the name of "career education." 
Second, there are serious questions regarding the relatively short time frames in which 
many of the studies have been conducted. Third, there are questions regarding the 
appropriateness of the measuring instruments used for assessing several of the "learner 
outcomes." Fourth, there are questions regarding appropriate models for use in evaluating 
career education. Finally, serious questions remain regarding the exact kinds of "treat- 
ment" given the "control groups" in many of the studies reported in this set of docu- 
ments. 

In hope of providing those charged with continuing and expanding efforts to evaluate 
career education efforts during the decade of the 1980s,' let us now turn our attention 
to what appear to be the most basic problems facing us in evaluating career education 
efforts. 

Evaluation of Career Education! Issues For The 1980s 
Issue I ! The Career Education "Treatment" 

The single most important issue to be resolved in designing and carrying out 
career education evaluation studies relate to the need to clarify and specify exactly 
what has been done in the name of "career education." A host of major problems 
must be faced if this issue is to be resolved, 

First, in the basic OCE policy monograph, A Primer For Career Education, the 
"career education treatment" is defined in terms of a total of 53 roles and functions 
to be carried out by eight different kinds of "actors" in the delivery of career educa- 
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tion. In order to meet responsibilities for Federal leadership while simultaneously 
avoiding any implications of possible Federal control, these 53 roles and functions 
were written in terms of what is to be done in career education without specifying 
how each role and function is to be carried out. In order for those who evaluate career 
education to specify, with a needed sense of exactness, exactly what is being done 
in the name of "career education," it will be necessary for them to specify the "how" 
as well as the "what" aspects of the career education treatment. In brief, simply to 
assert that the career education treatment, as specified in A Primer For Career Educa- 
tion, was carried out will not be sufficient. 

Second, it is both unreasonable and undesirable to attempt to demand that even 
the 53 general role and function statements found in the OCE definition of the "career 
education treatment" be carried out before one could say that "career education" is 
being practiced. It is unreasonable in view of the fact that, as with any aspect of 
Education, a number of schools of thought exist with respect to defining the nature 
and goals of "career education." Like the term "evaluation" itself, different experts 
assign differing meanings to the words. It is undesirable in that, if everyone approached 
the delivery of career education in the same basic way, opportunities for discovering 
new and better delivery approaches would be lost. Thus, the 53 roles and functions 
proposed by OCE must continue to be regarded as only one of several possible ways 
of defining the "career education treatment." Evaluators are, however, under a clear obli- 
gation to specify exactly what they mean by the term "the career education treat- 
ment." Too many,to date, have failed to accept this obligation. 

Third, if the rationale behind the 53 roles and functions to be performed by the 
eight sets of "actors" pictured in A Primer For Career Education is accepted, it is 
obvious that what is to be evaluated will be a community effort, not simply an effort 
of the Education system alone. There will surely be a host of problems associated 
with verifying the degree to which such a total community effort is present, let alone 
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evaluating its effectiveness. The data collection problems inherent In verifying the 
presence of the effort clearly point to a concern relative to the eyaluability of career 
education as a total community effort. Further, those who evaluate career education 
as a community collaborative effort will obviously have to do so based on concepts 
of community accountability for the success - or lack of success - of career educa- 
tion. It seems obvious that, to reach the stage in any community where this require- 
ment can be met, a given career education effort wilJ have to be In place for a period 
of several years. It would be completely unrealistic to attempt to evaluate career 
education in terms of these 53 roles and functions at the end, say, of only one year 
of operation. Still, some kind of evaluation of efforts at the end of even the first 
year will be called for. The model proposed in A Primer For Career Education will 
not be one that can be vallcfly applied at the end of the first year of operation. 

Fourth, if these 53 roles and functions are studied carefully, it will be obvious 
that each of them Is, itself, subject to evaluation. Further, it will be obvious that 
the proper ways In which most, if not all, of these 53 roles and functions should be 
evaluated will require evaluative criteria quite different in nature from the OCE 
"learner outcomes" that are proposed for use in evaluating the total career education 
effort. At this point in time, proper evaluation designs for use in evaluating many 
of these 53 roles and functions have not been devised. Until they are devised and 
put Into operation, It will be operationally impossible to evaluate the total "career 
education treatment" in terms of the OCE "learner outcomes." 

Issue 2t The Need For Intermediate Criteria 

A second basic issue facing those concerned with evaluation of career educa- 
tion is evident in the developmental longitudinal nature of the career education 
concept itself. The career education "treatment" Is one that is expected to begin 
no later than when the individual enters the formal Education system (and, hopefully, 
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even before that point in time in terms of home/family actions). It is expected to 
continue throughout the period of time the individual remains in the formal Education 
system. Further, it is expected to be a "treatment" that becomes available to persons 
at those times in their adult lives when they again have a need and/or desire to use 
Education as a means of helping to resolve work/education problems. In short, the 
career education "treatment" is seen, conceptually, as one that should be available 
to persons throughout almost all of the life cycle - including the retirement years. 

In order to reduce this broad, philosophical view to one that is more nearly sub- 
ject to systematic evaluation, OE's Office of Career Education has described "learner 
outcomes" for career education in terms of competencies and actions the individual 
should be expected to demonstrate at that point in time whenshe/he leaves the formal 
education system for the first time. This time of leaving may, in the case of high 
school dropouts, be prior to completion of the secondary school. For many persons, 
that point in time will be represented by high school graduation. For many others, 
It will be represented when one leaves a community college, a four year college/university 
system, or a graduate/professional school with an advanced degree. 

For purposes of narrowing still further specification of career education "learner 
outcomes" to those that can be associated specifically with the K-12 portion of the 
Education system, the original "learner outcomes" specified in the OE policy paper 
An Introduction To Career Education has been further refined and narrowed to the 
nine "learner outcomes" listed in OCE's latest basic career education policy paper, 
A Primer For Career Education. While, at the time that paper was written, these 
"learner outcomes" were still being defined as though they apply to persons leaving 
the Education system at any age or at any level, they have been put in a form that 
will allow their initial use to be made at the point when the individual leaves the K- 
12 system of Education. The 9 "learner outcomes" included in this more restrictive 



view Include; 

1. Competent in the basic academic skiils required for adaptability in our 
rapidly changing society 

2. Equipped with good work habits 

3. Equipped with a personally meaningful set of work values that foster in 
them a desire to work 

4. Equipped with career decision-making skills, job-hunting skills, and job- 
getting skills 

5. Equipped with a degree of self-understanding and understanding of 
educational-vocational opportunities sufficient for making sound 
career decisions 

6. Aware of means available to them for continuing and recurrent 
education 

7. Either placed or actively seeking placement in a paid occupation, in 
further education, or in a vocation consistent with their current 
career decisions 

8. Actively seeking to find meaning and meaningf ulness through work in 
productive use of leisure time 

9. Aware of means available to themselves for changing career options - 
societal and personal constraints impinging on career alternatives 

The basic contention made in A Primer For Career Education is that, IF the 
53 roles and functions outlined for the "career education treatment" are carried out 
properly DURING THE ENTIRE K-12 PERIOD OF SCHOOLING, each of these nine 
basic "learner outcomes" will be achieved. This is the basic contention that needs 
to be subjected to careful evaluation. To adequately test this contention will demand 
that data be collected for a given group of individuals, beginning at the Kindergarten 
level and continuing until graduation from high school, demonstrating that the com- 
plete "career education treatment" has, in fact, been adequately provided. Applica- 
tion of evaluation procedures for each of the nine "learner outcomes" should then be 
applied for persons who have received this verified "career education treatment." 

This kind of evaluation of the "career education treatment" has not vet been 
undertaken. Since the formal career education movement did not begin until around 
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1970, it is obviously too soon to expect that this could have occured. Theoretically, 
there should be at least a few K-12 school systems somewhere in the Nation who began 
their career education efforts in the 1970-72 period and have continued them ever 
since. If such school systems could be located - and IF they had available the kinds 
of data required for verifying that the 53 roles and functions outlined In the "career 
education treatment" have, in fact,been taking place for pupils ever since they began 
as Kindergarten enrolles in the 1970-72 period, it could be expected that the kinds 
of evaluation studies most needed could be conducted sometime during the 1982-84 
period. Realistically, this simply cannot be expected to occur both because (a) the 
career education concept was not fully developed during the 1970-72 period, and (b) 
no current K-12 school system has the kinds of detailed data required to verify 
the complete "career education treatment" has taken place. It seems more reason- 
able to believe that this kind of evaluation study probably will not be possible until 
sometime during the decade of the 1990s. 

In the meantime, we will have to content ourselves with the use of only a portion 
of the nine "learner outcomes" and, In many cases, with the use of Intermediate criteria 
rather than the ultimate criteria represented by the "learner outcomes" themselves. 
Further, we will have to recognize that, in the case of certain of these "learner out- 
comes," It would be grossly inappropriate to apply them to pupils at the elementary 
school level. This, in no way, means that no sound evaluations of career education's 
effectiveness can be conducted at this point in time. Rather, it simply means that 
the career education movement is still too young to be able to test Its effectiveness 
In attaining all nine of the "learner outcomes.' 1 Further, while we are not yet In a 
position to evaluate the effectiveness of career education over the entire K-12 period 
of Education, we are now in a position to begin the process of longitudinal evaluation 
studies extending over a period much longer than a single school year - or portloh of 
a single school year. The decade of the 1970s for career education was one characterized 
largely by short-term studies of the effectiveness of career education. The decade 
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of the 1980s will, hopefully, be one where evaluation studies cover a period of several 
years. If this can be done, then the decade of the 1990s can be one where evaluation 
studies cover the entire K-12 period of schooling. 

Issue 3; Finding and Using Proper Control Groups 

It Is obvious that a variety of kinds of evaluation designs are needed in the evalua- 
tion of career education efforts. It is equally obvious that, among such designs, many 
persons will still place their highest values on the traditional "experimental/control" 
model which calls for persons in the "experimental" groups to receive the "career 
education treatment" while persons In the "control groups" receive some other "treat- 
ment." Thus, If the traditional "experimental/control model 11 is not regarded as 
sufficient for demonstrating the effectiveness of career education, it will continue 
to be necessary. There are several problems involved here, none of which have been 
adequately recognized to date. 

First, it should be recognized that, In many of the currently existing evaluation 
studies, there has been an almost complete absence of the description of the "treat- 
ment 11 given to pupils in "control 11 groups. There exists as great a need to describe 
the "control treatment" as to describe, in exact terms, the "career education treat- 
ment. 11 It Is hoped that future evaluation studies will recognize this basic fact. 

Second, it should be recognized that, in a broad, generic sense, the "career 
education treatment" demands thatt (a) a sense of purposefuiness be inserted Into 
the teaching/learning process for both students and teachers - i.e., that students 
feel It Is important to learn what Is being taught and that teachers feel it is important 
to help students learn what Es being taught; and (b) the broader community be used 
as a learning resource In the total educational effort. In the "career education treat- 
ment," an emphasis on the Importance of education as preparation for work - paid 
and unpaid - as part of the person's total lifestyle is used as the basic vehicle for meet- 
Ing those demands. The essential questions to be resolved here are: (a) should "control 
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groups: consist of "treatments" where these two demands are not part of the "treat- 
ment"? or (b) should "control groups" consist of "treatments" where these two demands 
are met through some vehicle other than the "careers emphasis" of career education? 
If the first of these two alternatives Is adopted, one would have to assume that the 

teaching/learning process holds little meaning for either pupils or their teachers and 

$ 
that the education system has isolated itself from the broader community. While 

we know, of course, these conditions exist in many K-12 school systems today, to use 
these as "control groups" will surely be perceived by many as "stacking the cards" 
In favor of showing greater gains for "experimental groups" than for 'control groups." 
It may be the best thing to do, but it is almost certain to result in this kind of basic 
criticism. 

If, on the other hand, "control groups" are defined in terms of use of an alternative 
vehicle for: (a) making the teaching/learning process more meaningful to all teachers 
and all pupils at all grade levels; and (b) involving the broader community in the total 
educational process, then the "career education treatment" would have to be very 
powerful indeed if significant gains are to be shown for "experimental groups" as 
opposed to "control groups." Ideally, this would appear to be much preferable to the 
first alternative. The problem is one of Identifying such an alternative vehicle that 
uses an emphasis on something other than "careers" while employing either the same 
basic strategies or a set of different basic strategies than those used in career education. 

Those contemplating this alternative need to do so recognizing the basic strategies 
involved in the career education effort which have kept this effort going for 10 years 
- three times as long as the typical "educational reform" efforts that have come along 
In the past. Such strategies include: (a) promoting the effort based on a basic need 
perceived as real, Important, and continuing in nature - as seen by educators, students, 
parents, and the general public; (b) choosing a need that Is applicable to ALL pupils 
at ALL grade levels and within ALL subject matter disciplines in the education 
system) (c) choosing a need that influential community members will acknowledge 
as real and recognize as one for which their expertise would be helpful in meeting? 
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(d) choosing a need that involves "selling" rather than "buying" teachers - i.e., a need 
which, if met, will enhance the instructional goals of the teacher rather than being 
met through adding new tasks and responsibilities that may Interfere with attainment 
of instructional goals; (e) choosing a need that will not only gain parental support but 
will, in addition, give parents clear roles and responsibilities! (f) choosing a need which 
represents one of the basic goals of Education which, if met, will enhance attainment 
of other basic goals of Education as well; (g) choosinga need which can be met with- 
out greatly increased costs to the education system,- (h) choosing a need which will 
result In an effort clearly susceptible to clear evaluations; and (i) choosing a need 
which holds potential for becoming institutionalized as part of the basic fabric of the 
education system itself. By concentrating on the need to help ALL pupils better 
understand and solve problems of work/education relationships, career education has 
been able to use all of these strategies. To date, no such alternative strategy that 
also meets all these needs appears to be present. 

As a result, the "career education treatment" appears to be one best compared 
with whatever else the school system was doing before the "career education treat- 
ment" started. If "experimental" and "control" groups are established In such a school 
system, it is obvious that it will be relatively easier to demonstrate career education 
to be "effective" in those school systems where low quality educational efforts existed 
than in school systems where high quality educational efforts were in place prior to 
the time a 'career education treatment" was instituted. In any event, the "control 
treatment" Is fully as important to describe and define as Is the "career education 
treatment." 

Issue ft: The Teachln^/Learnlng Process 

The teaching/learning aspect of the "career education treatment" represents 
a sufficiently important part of the previous Issue so as to deserve as additional 
separate discussion. 



In A Primer For Career Education, a total of 10 roles and responsibilities are 
assigned classroom teachers in the total "career education treatment." The follow- 
ing portion of these 10 roles and responsibilities are of concern here: 

"1. Seeking to improve academic achievement through using a "careers 
emphasis" as a vehicle to: 

a. Introduce a sense of purposefulness and meaningfulness into the 
teaching/learning process for both student and teacher through 
emphasizing that one of the reasons for learning the subject 
matter is that people use it in their work. 

b. Use a positive approach with students through rewarding students for 
what they have accomplished rather than emphasizing what they have 
failed to accomplish, how much more they have to accomplish, or how 
many other students accomplished more. 

c. Introduce variety into the teaching/learning process through utilizing 
the personnel and physical resources of the broader community as 
vehicles for improving student achievement - for emphasizing that 
students can learn in more ways than simply through reading, in more 
places than the classroom, and from more persons than certified class- 
room teachers. 

2. Consciously and conscientiously provide rewards to students who exhibit 

and practice such basic good work habits as: (a) coming to work (to 
school) on time; (b) completing assignments that are begun) (c) doing the 
best that they canj and (d) coooerating with one's fellow workers (students) 

3. Combining a cognitive and experiential approach in the teaching/learning 
process through emphasizing the dual desirability of "doing to learn" and 
"learning to do" 

<(. Helping students acquire decision-making skills through using a project 
activity-oriented approach, when appropriate, in the teaching/learning 
process that allows students to actually engage in the decision-making pro- 
cess 

5. Systematically attempting to reduce biases students may have with respect 
to race, sex, or handicapping conditions in ways that will maximize freedom 
of choice for all persons. 

6. Helping students discover ways in which the subject matter being learned 
can be valuable to students in productive use of leisure time." 

The remaining four "roles and responsibilities" assigned classroom teachers in the "career 
education treatment' are more specifically oriented around the "careers" emphasis. 
The six listed above, while Involving use of a "careers emphasis" as a vehicle, quite 
obviously pertain to a generic set of activities which good teachers have always 



practiced. This is undoubtedly one of the reasons some of career education's critics 
have complained that all career education advocates are asking for is "good teach- 
ing." 

The question, in evaluating career education through the use of "experimental" 
and "control" groups, is one of whether or not, for pupils in the "control treat- 
ment," these six generic kinds of teacher activities are to be actively discouraged, 
It this were to happen, the net result might well be that the resulting evaluations 
would simply become a matter of comparing "good teaching" with "poor teaching," 
This would not appear to be a fruitful course to follow - in spite of the fact that it 
would obviously make "career education' r look good. The only viable solution that 
I can see is to depend on comparing the "career emphasis" as a vehicle for producing 
good teaching with whatever other vehicles are being used by "good teachers" in the 
"control groups." Once again, evaluators of career education are faced with a 
situation where It will be much more difficult to produce positive evidence of the 
effectiveness of career education in schools where "good teachers" already exist 
than in schools where they don't, This is simply something that must be recognized 
and faced. 

This problem points up clearly the Importance of conducting career education 
evaluations over a period of years with longitudinal data rather than in a period of 
a single school year- or portion of a school year. Any procedure designed to improve 
academic achievement is likely to be faced with this same problem - i.e., real gains 
in academic achievement are not likely to occur with, a very short time period. 

Concluding Remarks 

The career education effort has been subjected to a wide variety of kinds of 
evaluation efforts during the decade of the 1970s - its first real decade of existence, 
On the whole, results of those evaluative efforts completed to date have been more 



positive than negative. There is more reason to believe that career education is ef- 
fective than there is to believe that it is lacking in effectiveness. 

At the same time, the decade of evaluative efforts carried out during the 1970s 
has made clear some of the major problems facing career education in demonstrating 
both its worth and its effectiveness. Some of those problems stem from the fact that 
the career education concept is still evolving - thus making any kind of final statement 
with respect to the "career education treatment" an unattainable goal. A part of the 
general problem facing career education is its longitudinal, developmental nature that 
is going to demand longitudinal studies before its true effectiveness can be assessed. 
Coupled with this is an obvious need to evaluate the presence of the "career education 
treatment" itself as well as the effects of that treatment. This means that a variety 
of evaluation designs must be put into operation. 

Finally, the greatest challenge facing those charged with evaluation of career 
education efforts will be that of specifying exactly what is done in the name of the 
"career education treatment" and what is done with groups used for comparison pur- 
poses. This is a crucial problem to be solved if the goat of retaining local control over 
career education efforts is to be maintained. The goal of working toward local autonomy 
and away from any form of Federal control is worthy enough so as to justify whatever 
effort Is required to solve this problem. 
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